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EXECUTIVE  SUMMARY 

This  test  report  presents  the  results  for  the  Auxiliary  Tasking  during  Boundary 
Avoidance  Tracking  (AT  BAT)  Test  Management  Project  (TMP).  The  AT  BAT  test 
team  from  the  USAF  Test  Pilot  School  (TPS)  at  Edwards  AFB,  CA  performed  testing  to 
research  pilot  behavior  in  the  presence  of  boundaries. 

The  AT  BAT  TMP  was  conducted  at  the  request  of  USAF  TPS  in  collaboration 
with  AFIT.  The  Commandant  of  USAF  TPS  directed  this  program.  All  testing  was 
accomplished  under  TPS  Job  Order  Number  MT080500.  Two  calibration  sorties  and  ten 
data  sorties  were  flown  on  the  Variable-stability  In-flight  Simulator  Test  Aircraft 
(VISTA)  NF-16D  aircraft,  between  5  September,  2008,  and  23  September,  2008,  totaling 
16.5  flight  hours.  Additionally,  approximately  20  hours  of  simulation  time  were 
accomplished  in  the  Large  Amplitude  Multi-Mode  Aerospace  Research  Simulator 
(LAMARS)  motion-based  simulator  at  the  Air  Force  Research  Laboratory  at 
Wright-Patterson  AFB,  OH. 

This  testing  investigated  the  effects  of  imposing  boundaries  on  a  dynamic  point 
tracking  task  in  flight.  Data  collected  during  the  flights  in  the  LAMARS  simulator  and 
the  VISTA,  both  programmed  to  simulate  three  aircraft  with  different  handling  qualities, 
were  compared  with  the  theoretical  BAT  model  in  order  to  detennine  the  overall  theory 
validity.  In  the  tests,  the  pilot  followed  a  point  tracking  task  profile  (a  moving  roll-angle 
target)  displayed  in  the  Heads-Up  Display  (HUD).  The  point  tracking  task  profile  also 
had  boundaries,  or  limits,  displayed  in  the  HUD.  As  the  tracking  task  progressed,  the 
boundaries  shrank  in  discrete  increments.  The  boundary  avoidance  theory  stated  that  as 
the  boundaries  encroached  on  the  tracking  task,  the  pilot  would  transition  to  a  period  of 
boundary  awareness,  where  the  shrinking  boundaries  would  alter  task  performance. 
Eventually,  as  the  boundaries  got  even  closer,  the  pilot  would  transition  to  tracking  the 
boundaries  and  abandon  the  point-tracking  task  entirely. 

Previous  BAT  flight  test  research  had  focused  on  longitudinal  maneuvering.  This 
test  program  instead  used  lateral-directional  maneuvering  in  the  fonn  of  a  roll-angle 
tracking  task.  In  addition,  this  program  implemented  a  secondary  task  (displayed  in  the 
HUD  and  tracked  with  a  button  on  the  throttle)  as  an  objective  measure  of  pilot  workload. 

The  data  from  this  test  showed  that  pilots  do  react  to  boundaries.  The  average 
stick  force  and  stick  actuation  rate  tended  to  increase  as  the  boundaries  got  closer,  even 
though  the  underlying  point  tracking  task  did  not  change.  In  addition,  the  roll  error  (as  a 
percentage  of  boundary  size)  increased  dramatically  at  the  lowest  boundary  size, 
suggesting  that  the  point  tracking  task  had  been  abandoned  entirely.  The  theoretical  BAT 
model  was  compared  to  pilot  behavior  during  the  tests,  and  was  shown  to  match  pilot 
inputs  relatively  well  during  periods  of  suspected  boundary  avoidance.  Finally,  the 
smallest  achievable  boundary  size  and  the  secondary  task  performance  in  a  given  aircraft 
were  shown  to  correlate  with  the  Cooper-Harper  ratings  given  to  that  aircraft. 
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Introduction 

General 

This  Test  Management  Project  (TMP)  explored  the  Boundary  Avoidance 
Tracking  (BAT)  concept.  The  first  objective  of  the  Auxiliary  Tasking  during  BAT  (AT 
BAT)  program  was  to  observe  pilot  behavior  during  a  tracking  task  and  secondary  task  in 
the  presence  of  boundaries,  and  compare  this  behavior  to  the  behavior  predicted  by  the 
BAT  theory.  A  second  objective  was  to  search  for  a  correlation  between  the  subjective 
pilot  opinion  ratings  of  the  aircraft  from  a  Cooper-Harper  task  and  the  objectively- 
measured  perfonnance  and  workload  of  a  separate  task  in  the  roll  axis. 

The  AT  BAT  TMP  was  conducted  at  the  request  of  the  Air  Force  Test  Pilot 
School  (TPS)  in  collaboration  with  the  Air  Force  Institute  of  Technology  (AFIT),  Wright 
Patterson  AFB,  OH.  The  Commandant  of  USAF  TPS  directed  the  program.  All  testing 
was  accomplished  under  TPS  Job  Order  Number  (JON)  MT080500.  It  consisted  of  12 
test  missions  totaling  16.5  flight  hours  from  05  September,  2008,  to  23  September,  2008, 
using  the  NF-16D  Variable-stability  In-flight  Simulator  Test  Aircraft  (VISTA). 

Background 

Previous  pilot  models,  dating  back  as  early  as  World  War  II,  made  the  assumption 
that  the  pilot  produces  control  actions  based  on  observed  error  between  a  desired  state 
and  an  actual  state,  be  it  regulation  (e.g.  keep  the  wings  level)  or  tracking  (e.g. 
climb/descend/tum  to  the  appropriate  altitude/heading).  This  type  of  control  will 
subsequently  be  labeled  point  tracking. 

In  2004,  Mr.  William  Gray  at  the  US  Air  Force  Test  Pilot  School  introduced  the 
concept  of  Boundary  Avoidance  Tracking,  challenging  the  assumption  that  a  pilot  only 
works  to  maintain  certain  conditions.  He  theorized  that  a  pilot  often  acts  to  avoid  a 
specific  condition,  or  a  boundary.  An  analogy  to  illustrate  the  difference  involves  a 
bicyclist  maintaining  a  track  along  a  line,  such  as  a  painted  line  on  a  road.  Most  people 
would  have  no  trouble  maintaining  such  a  track.  However,  superimpose  this  white  line 
across  the  Grand  Canyon  and  the  task  becomes  significantly  more  difficult.  Most  people 
would  be  well  aware  of  the  boundaries  of  their  task,  since  crossing  them  would  have  fatal 
ramifications.  Mr.  Gray’s  research  indicated  that  in  such  a  scenario,  the  subject  has  lost 
awareness  of  the  centerline  altogether,  using  control  solely  to  avoid  the  boundaries. 

The  next  step  in  the  development  of  this  model  was  to  attempt  to  match  the  model 
to  experimental  data.  To  do  so,  subjects  flew  a  joystick-based  desktop  simulator  in  which 
the  object  was  to  follow  a  tracking  bar  moving  up  and  down  on  the  screen,  while 
remaining  within  decreasing  boundaries  on  either  side  of  the  tracking  bar.  Subjects  were 
instructed  to  treat  the  boundaries  as  deadly;  the  task  in  fact  stopped  when  a  boundary  was 
exceeded.  During  the  simulations,  the  operators  had  little  problem  following  the  tracking 
bar  when  the  boundaries  were  far  apart,  but  had  increasing  difficulty  tracking  the  bar  as 
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the  boundaries  closed  in,  often  resulting  in  a  pilot-in-the-loop  oscillation  (PIO)  leading  to 
boundary  excursion.  The  data  were  analyzed  during  segments  of  suspected  BAT  activity 
(especially  during  the  PIOs),  and  the  BAT  model  predicted  pilot  inputs  relatively  well 
(reference  1). 

The  model  was  then  explored  during  flight  test.  In  2005,  TPS  conducted  flight 
tests  in  project  HAVE  BAT  with  T-38s  to  observe  the  BAT  phenomenon  in  flight, 
refining  the  model’s  ability  to  predict  pilot  inputs  (reference  2).  The  test  aircraft  was 
required  to  fly  close  fonnation  with  a  target  aircraft  maneuvering  longitudinally  while 
maintaining  position  within  certain  visual  references.  An  aircraft-mounted  camera  was 
placed  in  the  test  aircraft  to  record  these  visual  references  for  use  in  data  analysis.  The 
data  proved  very  difficult  to  extract  from  the  video  and  severely  limited  BAT  model 
characterization.  However,  the  tests  were  successful  in  demonstrating  BAT  in  flight  and 
in  suggesting  improvements  for  future  flight  tests. 

In  2006,  TPS  conducted  another  series  of  flight  tests  in  project  BAT  DART,  this 
time  using  the  NF-16D  VISTA  (reference  3).  The  VISTA  solved  the  problems  inherent 
in  the  T-38  camera  system  by  displaying  a  tracking  task  and  boundaries  in  the  Heads-Up 
Display  (HUD),  thereby  eliminating  the  need  for  a  target  aircraft  and  video  cameras.  Data 
collection  was  instead  accomplished  with  on-board  software  and  data  recording  devices. 
The  use  of  VISTA  also  permitted  the  simulation  of  four  different  aircraft-response 
models  in  one  aircraft  during  the  same  flight.  The  primary  focus  of  BAT  DART  was 
correlating  pilot  opinion  ratings  of  the  different  simulated  aircraft  with  tracking 
perfonnance  in  the  presence  of  boundaries.  The  tests  were  successful  in  demonstrating  a 
qualitative  relationship  between  pilot  opinion  ratings  and  both  the  length  of  time  before 
the  boundaries  were  exceeded  and  the  average  tracking  error. 

Previous  tests  involved  only  longitudinal  maneuvering.  The  tracking  tasks 
involved  in  these  tests  subjected  the  test  pilots  to  large  changes  in  normal  acceleration 
(g),  including  large  negative  load  factors.  These  changes  in  g  were  thought  to  influence 
pilot  behavior,  and  perhaps  be  the  source  of  an  additional  (and  unwanted)  boundary.  This 
test  program  used  a  roll-angle  task  in  order  to  eliminate  this  additional  boundary  and 
possibly  provide  “cleaner”  data. 

The  previous  attempt,  in  the  BAT  DART  TMP  (reference  3),  to  correlate  BAT 
perfonnance  and  Cooper-Harper  ratings  used  only  task  perfonnance  in  the  attempted 
conelation.  However,  Cooper-Harper  ratings  require  assessment  of  both  perfonnance 
and  workload.  Using  a  secondary,  non-control  task  as  a  quantitative  measure  of  pilot 
workload  (i.e.  pilot  spare  capacity)  was  used  to  solve  this  problem.  This  test  added  an 
additional,  secondary  task  to  the  roll  tracking  task.  The  task  involved  actuating  a  switch 
in  response  to  HUD  symbology  that  changed  in  an  unpredictable  manner.  In  addition, 
eye -tracking  equipment  (available  for  use  during  simulator  tests)  provided  information 
about  pilot  attention  and  workload  during  the  profiles. 

The  tests  were  divided  into  three  profiles.  The  first  profile  required  tracking  a  roll 
angle  for  two  minutes  with  defined  performance  criteria.  The  purpose  of  this  profile  was 
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to  assign  a  Cooper-Harper  Rating  to  a  given  aircraft  model.  The  second  task  required 
tracking  a  roll  angle  in  the  presence  of  boundaries.  The  purpose  of  this  profile  was  to 
record  pilot  behavior  during  a  tracking  task  in  the  presence  of  boundaries,  and  to  provide 
a  baseline  for  comparison  with  the  secondary  task  profile.  No  Cooper-Harper  rating  was 
assigned  at  the  end  of  this  task.  The  last  profile  required  tracking  a  roll  angle  in  the 
presence  of  boundaries,  with  the  additional  task  as  described  above.  The  purpose  of  this 
profile  was  to  record  pilot  behavior  during  a  tracking  task  in  the  presence  of  boundaries 
and  a  secondary  task.  The  perfonnance  of  the  secondary  task  provided  an  indication  of 
pilot  workload  (i.e.  “spare  capacity”  during  the  primary  roll-angle  tracking  task).  No 
Cooper-Harper  rating  was  assigned  at  the  end  of  this  task. 

Fixed-base  simulation  was  accomplished  as  part  of  an  in-progress  AFIT  Master’s 
thesis  at  AFRL.  These  simulations  were  structured  very  closely  to  the  tests  performed 
during  this  test  program.  The  tests  used  a  roll-angle  tracking  task  displayed  in  the  HUD, 
and  a  secondary  task  consisting  of  four  arrows  which  illuminated  one-at-a-time  in  a 
Heads  Down  Display.  The  primary  task  performance  (measured  as  minimum  boundary 
size  achievable)  was  shown  to  worsen  as  the  aircraft  handling  qualities  worsened.  The 
secondary  task  performance  was  shown  to  remain  approximately  constant  until  a  certain 
boundary  size  (dependent  on  the  aircraft  handling  qualities)  was  reached,  at  which  point 
perfonnance  decreased.  This  point  was  labeled  the  “boundary  awareness  point”.  A 
combination  of  primary  and  secondary  performance  was  shown  to  conelate  well  with 
Cooper-Harper  ratings. 

Motion-based  simulation  was  also  accomplished  by  the  student  test  team  at  the 
full-motion  Large  Amplitude  Multi-Mode  Aerospace  Research  Simulator  (LAMARS) 
simulator  at  AFRL.  A  draft  of  the  flight  test  plan  was  implemented  in  the  simulator,  with 
improvements  incorporated  into  the  flight  test  plan. 

Program  Chronology 

A  joint  Technical  Review  Board/Safety  Review  Board  (TRB/SRB)  was 
conducted  on  25  Aug  08.  The  TRB  was  chaired  by  Mr.  Gary  Aldrich,  USAF  TPS/EDT. 
The  SRB  was  chaired  by  Mr.  Rob  Warner  of  AFFTC/SET. 

The  test  project  consisted  of  three  full  days  of  simulator  studies  at  the  LAMARS, 
a  ground  checkout,  and  twelve  NF-16D  VISTA  sorties  totaling  16.5  flight  hours  within 
the  R-2508  complex  at  Edwards  Air  Force  Base,  California.  The  simulator  studies 
occurred  on  16-18  July  2008.  Ground  tests  occurred  on  03  and  04  September  2008. 
Flight  testing  occurred  from  05  to  23  September  2008. 

Test  Item  Description 

The  NF-16D  VISTA  was  a  modified  Block  30  F-16D,  based  on  the  Israeli  Air 
Force’s  airframe  design.  The  avionics  were  Block  40  configuration,  and  a  custom  Digital 
Flight  Control  Computer  (DFLCC)  was  installed.  The  Variable  Stability  System  (VSS) 
was  a  five-degree-of-freedom  system  used  to  simulate  various  types  of  aircraft  handling 
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qualities  by  changing  the  static  and  dynamic  flight  characteristics.  The  front  seat  was  the 
simulation  cockpit  and  the  rear  seat  had  the  primary  flight  controls. 

With  the  VSS  engaged,  the  Evaluation  Pilot  (EP)  in  the  Front  Cockpit  (FCP) 
could  control  the  aircraft  with  either  a  centerstick  or  sidestick,  using  the  programmed 
control  laws.  This  project  used  the  sidestick  controls.  As  a  safety  feature,  the  Safety 
Pilot  (SP)  in  the  Rear  Cockpit  (RCP)  maintained  pilot-in-command  features.  All  displays 
were  programmed  to  the  user’s  needs. 

Specific  modifications  to  the  NF-16D  aircraft  for  the  AT  BAT  project  include: 

1)  Custom  VSS  programming  to  simulate  3  different  aircraft’s  lateral/directional 
control  laws. 

2)  Custom  display  programming  to  input  a  target  and  boundary  profile  into  the 
HUD,  as  well  as  a  secondary  task  in  the  lower  left-hand  comer,  as  shown  in 
figure  1. 


Figure  1:  Sample  AT  BAT  HUD  Symbology 


The  VISTA  used  HUD  and  VSS  video  tapes  to  record  data  during  flight. 

Calspan,  which  was  in  charge  of  VISTA  operation,  designed  three  aircraft 
models  for  this  test.  The  intent  was  to  make  models  with  varying  levels  of  degraded 
lateral-directional  handling  qualities.  See  the  aircraft  model  development  section  and 
appendix  A  for  specific  information  about  the  models  and  the  development  procedure. 
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Overall  Test  Objective 

The  primary  objective  of  the  AT  BAT  program  was  to  observe  pilot  behavior 
during  a  tracking  task  and  secondary  task  in  the  presence  of  boundaries  and  to  compare 
this  behavior  to  the  behavior  predicted  by  the  BAT  theory.  A  second  objective  was  to 
investigate  a  correlation  between  the  subjective  pilot  opinion  ratings  of  the  aircraft  from  a 
Cooper-Harper  task  and  the  objectively-measured  perfonnance  and  workload  of  a 
separate  task  in  the  roll  axis.  All  objectives  were  met. 
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Test  and  Evaluation 

Flight  test  was  accomplished  from  05  to  23  September,  2008.  Twelve  sorties 
were  flown  for  16.5  flight  hours  inNF-16D  86-00048. 

The  first  two  flights  were  used  to  ensure  the  aircraft  models  provided  by  Calspan 
were  controllable  and  provided  a  sufficient  handling  qualities  spread,  in  tenns  of  their 
ability  to  satisfactorily  accomplish  the  roll  angle  tracking  task.  Each  configuration  was 
tested  along  with  programmed  test  inputs  applied  by  the  VSS. 

Data  for  the  AT  BAT  TMP  were  collected  on  the  remaining  ten  flights.  Each 
member  of  the  test  team  flew  twice.  Flight  test  engineer  data  was  included  in  the  analysis 
and  showed  no  bias  to  lack  of  experience.  The  profile  order  was  randomized  between 
flights  and  within  flights  to  preclude  the  test  subject  anticipating  and/or  learning  what  the 
next  configuration  was. 

Prior  to  each  test,  the  aircraft  was  trimmed  in  straight  and  level,  unaccelerated 
flight  at  the  prescribed  flight  condition  of  300  KCAS  and  20,000  ft  PA.  The  evaluation 
pilot  initiated  the  profile  using  a  button  on  the  control  stick. 

Three  types  of  profiles  were  flown  on  each  sortie. 

For  the  Cooper-Harper  task  (figure  2),  the  roll-angle  target  was  displayed  in 
the  HUD.  The  pilot  attempted  to  keep  the  target  aligned  with  a  caret  at  the  top 
of  the  HUD.  A  set  of  lines  fixed  at  ±  5  degrees  from  the  target  defined  the 
boundaries  of  desired  perfonnance,  and  a  set  of  lines  fixed  at  ±  7.5  degrees 
from  the  target  defined  the  boundaries  of  adequate  performance.  The  pilot 
assigned  a  Cooper-Harper  rating  (see  appendix  C)  based  on  perfonnance  and 
workload  during  the  profile. 


aiming  reference 

small  lines  =  desired 
large  lines  =  adequate 


Figure  2:  Cooper-Harper  Task  HUD  Symbology 
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For  the  BAT  task  (figure  3),  the  roll-angle  target  was  displayed  as  in  the 
previous  profile.  The  5  degree  and  7.5  degree  lines  from  the  previous  profile 
were  replaced  with  a  single  set  of  boundaries  starting  at  ±  30  degrees.  The 
boundaries  remained  constant  for  30  seconds,  and  shrank  by  approximately  6 
degrees  at  the  end  of  the  30  second  period.  The  boundaries  continued  to 
shrink  by  approximately  6  degrees  every  30  seconds  until  the  end  of  the 
profile.  (Note  that  the  boundary  profile  requested  by  the  test  team  began  at 
±40  degrees  and  shrank  by  20  percent  every  30  seconds.  Due  to  programming 
anomalies,  the  actual  boundary  profile  implemented  in  the  VISTA  is  as 
described  above.)  The  pilot  treated  the  boundaries  as  life-threatening,  and 
maneuvered  as  required  to  prevent  exceeding  them.  The  profile  stopped 
automatically  when  the  fixed  caret  at  the  top  of  the  HUD  exceeded  the 
boundaries  for  0.5  seconds.  The  purpose  of  the  0.5  second  delay  was  to 
gather  data  for  pilot  behavior  outside  given  boundaries. 


Figure  3:  BAT  Task  HUD  Symbology 

For  the  BAT  plus  secondary  task  (figure  4),  all  procedures  were  the  same  as 
the  previous  profile,  except  that  a  secondary  task  was  displayed  in  the  HUD. 
A  set  of  four  arrows  illuminated  one  at  a  time,  and  the  pilot  deflected  the  radar 
cursor  switch  on  the  throttle  in  response  to  the  illuminated  arrow.  The  arrows 
consisted  of  a  single-headed  arrow  pointing  up,  a  double-headed  arrow 
pointing  up,  a  single-headed  arrow  pointing  down,  and  a  double -headed  arrow 
pointing  down  (see  figure  5).  The  cursor  control  was  deflected  up  once  for 
the  single-headed  arrow  pointing  up,  and  it  was  deflected  twice  for  the 
double -headed  arrow  pointing  up.  The  switch  was  deflected  down  once  for 
the  single -headed  arrow  pointing  down,  and  it  was  deflected  twice  for  the 
double -headed  arrow  pointing  down.  As  before,  the  pilot  treated  the 
roll-angle  tracking  task  as  the  primary  task,  and  treated  the  boundaries  as  the 
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primary  threat.  The  secondary  task  was  neglected  as  required  to  prevent 
exceeding  the  boundaries. 


Figure  4:  BAT  Plus  Secondary  Task  HUD  Symbology 


nu 

Figure  5:  Secondary  Task  Arrows 

The  first  profile  of  each  sortie  was  a  30  second  Cooper-Harper  roll-tracking  task 
using  a  Level  1  configuration  flown  as  a  warm-up  to  acquaint  the  subject  with  the 
symbology.  Subsequent  profiles  were  selected  using  a  randomized  test  matrix  (see 
appendix  B).  Within  the  matrix  were  a  2-minute  Cooper-Harper  task,  a  BAT  task,  and  a 
BAT  task  with  a  secondary  task  for  each  of  the  aircraft  models.  Thus,  each  sortie 
included  ten  total  profiles,  nine  of  which  were  data  profiles. 

Aircraft  Model  Development 

Calspan,  which  was  in  charge  of  the  test  aircraft  operation,  designed  three  aircraft 
models  for  this  test.  The  intent  was  to  make  Class  IV,  Phase  A  aircraft  models  with 
Level  1,  Level  2,  and  Level  3  lateral-directional  handling  qualities,  respectively.  The 
models  were  developed  using  roll  mode  and  Dutch  roll  parameters  from  MIL-STD- 
1797B  for  a  Category  IV  aircraft  in  flight  phase  A  (reference  4).  For  the  proposed  Level 
1  aircraft,  the  roll  mode  time  constant  and  Dutch  roll  frequency  and  damping  were 
chosen  from  the  suggested  Level  1  values  in  the  MIL-STD.  Likewise,  the  proposed 
Level  2  aircraft  had  a  roll  mode  time  constant  and  Dutch  roll  frequency  and  damping  that 
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each  were  predicted  to  result  in  Level  2  handling  qualities.  The  proposed  Level  3  aircraft 
was  designed  in  the  same  way.  See  appendix  A  for  specific  model  descriptions. 

The  proposed  models  were  tested  in  the  Large  Amplitude  Multi-Mode  Aerospace 
Research  Simulator  (LAMARS)  simulator,  and  did  not  have  the  desired  spread  in 
handling  qualities.  The  three  models  had  noticeably  different  handling  qualities,  but  were 
each  (in  their  own  way)  approximately  Level  2  aircraft.  The  data  from  the  LAMARS 
simulator  were  useful  for  this  test  program  in  that  it  showed  the  models  should  be 
modified  to  provide  a  larger  spread  in  handling  qualities. 

Additionally,  the  Cooper-Harper  task  was  adjusted  in  an  attempt  to  allow  for  a 
spread  in  perfonnance  and  workload  that  would  result  in  Levels  1,  2  and  3.  This  was 
done  by  adjusting  the  percentages  for  desired  and  adequate  criteria  until  the  level  of 
difficulty  and  workload  for  the  task  was  appropriate  to  allow  for  a  spread  in  Cooper- 
Harper  ratings. 

As  a  result  of  the  LAMARS  simulations,  Calspan  modified  the  aircraft  models  to 
attempt  a  larger  spread  in  handling  qualities.  These  new  models  were  implemented  in  the 
VISTA.  Due  to  uncertainty  in  the  VISTA  aerodynamic  model,  the  handling  qualities 
during  the  calibrations  flights  were  dramatically  different  than  predicted.  The  first 
calibration  sortie  showed  that  the  proposed  Level  2  and  Level  3  aircraft  were  actually 
uncontrollable.  Calspan  made  changes  to  the  models,  and  the  second  calibration  sortie 
demonstrated  some  improvement,  but  the  proposed  Level  3  aircraft  was  still 
uncontrollable. 

After  the  calibration  sorties,  the  team  decided  that  too  many  variables  were 
changing  among  the  models,  and  that  a  model  with  parameters  that  were  separately 
predicted  to  be  Level  3  would  in  combination  lead  to  an  uncontrollable  aircraft.  As  a 
result,  the  final  models  were  all  given  roughly  the  same  Dutch  roll  characteristics,  and  the 
roll  mode  time  constant  was  used  to  change  the  models.  The  achieved  handling  qualities 
of  the  models  are  described  below  in  the  Data  Analysis.  The  three  models  were  given  the 
names  “Aircraft  X”,  Aircraft  Y”,  and  “Aircraft  Z.”  See  appendix  A  for  specific  model 
descriptions. 

Secondary  Task  Development 

The  test  project  team  designed  two  secondary  task  profiles  for  this  test.  It  was 
expected  that  the  performance  in  the  secondary  task  would  decrease  as  boundaries  shrink, 
to  the  point  where  the  pilot  would  abandon  altogether  the  secondary  task.  The  task 
included  four  arrows  illuminated  on  the  lower-left  corner  of  the  HUD  one  at  a  time  and 
changed  every  2  to  4  seconds. 

In  the  LAMARS  simulator  sessions,  the  team  also  added  an  eye -tracker  system  to 
record  pilots’  eye  focal  point  when  flying  with  secondary  tasks.  The  eye-tracker  was  not 
cleared  for  flight  test  and  not  used  in  the  VISTA  sorties.  The  data  included  a  series  of  24 
frames-per-second,  MPEG-4  digital  video  in  752x480  pixel  resolution.  The  eye-tracker 
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used  a  Charge-Coupled  Device  (CCD)  to  track  the  center  of  pilot’s  right  pupil  and 
another  CCD  to  record  in  the  direction  the  pilot’s  head  was  pointed.  The  equipment  then 
superimposed  a  red  cross  (as  the  center  of  pupil)  on  the  front  view  video  data. 
Eye-tracker  performance  was  limited  by  its  lens  field-of-view  (roughly  ±15  degrees), 
CCD  sensitivity,  and  bore-sight  alignment.  Additionally,  when  the  pilot  blinked,  the 
eye -tracker  would  lose  track  and  the  cursor  would  jump  back  to  the  origin.  Even  with 
these  limitations,  the  team  still  gathered  representative  data  to  show  aircraft  with  varying 
levels  of  handling  qualities  affected  the  crosscheck  performance  of  the  pilot. 

Figure  6  shows  a  sample  of  a  single  pilot’s  secondary  task  crosscheck  for  each 
aircraft  model.  A  crosscheck  was  defined  as  a  movement  of  the  pilot’s  eye  from  the 
primary  task  to  the  secondary  task.  The  crosscheck  frequency  was  defined  as  the  number 
of  crosschecks  divided  by  the  time  between  boundary  changes  (30  seconds).  Decreasing 
crosscheck  meant  that  the  pilot  was  looking  at  the  secondary  task  less  often.  As 
expected,  the  crosscheck  ability  of  the  pilot  decreased  as  the  boundaries  decreased,  and 
also  decreased  in  proportion  to  decreasing  handling  qualities. 
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Figure  6:  Pilot  Crosscheck  Ability  (Eye-Tracker  Data),  Typical 

The  crosscheck  frequency  also  helped  validate  the  secondary  task  as  a  measure  of 
pilot  spare  capacity.  Figure  7  shows  a  typical  pilot’s  crosscheck  ability  compared  to  his 
secondary  task  performance.  The  trends  of  the  two  sets  of  data  followed  each  other 
closely,  which  suggested  that  the  secondary  task  alone  could  be  used  in  the  VISTA  as  a 
measure  of  pilot  spare  capacity  (since  the  eye-tracker  was  not  cleared  for  flight  test). 
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Figure  7:  Pilot  Crosscheck  Ability  Compared  with  Secondary  Task  Error,  Typical 

The  Calspan  software  developer  indicated  that  they  could  only  program  the 
secondary  task  switch  for  two  directions  in  the  VISTA,  so  the  tasks  were  changed  to 
accommodate  this  limitation  (see  the  beginning  of  the  test  and  evaluation  section  for  final 
symbology).  The  test  team  agreed  that  this  would  not  change  the  difficulty  of  the 
secondary  tasks,  and  thus  would  not  affect  their  use  as  measures  of  pilot  workload. 

Data  Analysis:  Pilot  Behavior  in  the  Presence  of  Boundaries 


The  first  objective  of  the  AT  BAT  project  was  to  observe  pilot  behavior  in  the 
presence  of  boundaries  to  determine  if  the  presence  of  those  boundaries  altered  the  pilot’s 
behavior  in  any  manner.  An  examination  of  the  pilot’s  stick  inputs  as  a  function  of  time 
shows  that  the  pilot  does  indeed  change  his  behavior  as  a  function  of  decreasing 
boundaries.  For  the  BAT  and  BAT  plus  secondary  tasks,  the  only  part  of  the  task  that 
was  changing  with  time  was  the  boundary  size.  The  roll  task  itself  was  constant 
throughout  the  time  history.  As  illustrated  in  figure  8,  the  force  the  pilot  input  on  the 
stick  increased  with  time.  This  was  typical  of  most  of  the  pilots’  force  trends  in  both 
LAMARS  and  the  VISTA.  The  only  part  of  the  task  that  was  changing  was  the 
decreasing  boundaries  (indicated  on  the  plot  in  the  stairstep),  indicating  that  the  pilot  used 
greater  force  on  the  stick  as  the  boundaries  became  tighter. 


12 


December  2008 


AT  BAT 


Example  of  Average  Stick  Force  Increase  as  Boundaries  Decreased 
Altitude:  20,000  ft  PA  Airspeed:  300  KIAS 

Ai  rcraft:  LA  MARS  Date:  1 8  Ju  1 2008 

Project:  AT  BAT 


Figure  8:  Time  History  of  Stick  Force  Input,  Typical 


This  changing  pilot  behavior  was  also  analyzed  through  the  concept  of  duty  cycle 
and  aggressiveness  (reference  5).  This  concept  was  a  way  of  looking  at  pilot  “gain”,  by 
describing  how  often  and  how  aggressively  the  inceptor  was  moving.  Duty  cycle  was 
defined  as  the  percentage  of  time  that  the  inceptor  was  moving  above  a  defined  threshold 
(set  at  5  pounds  per  second).  Aggressiveness  was  defined  as  the  average  rate  of  inceptor 
movement.  A  high  duty  cycle  and  low  aggressiveness,  for  example,  would  suggest  that 
the  pilot  was  constantly  moving  the  inceptor,  but  at  small  rates. 

Average  duty  cycle  and  aggressiveness  were  calculated  for  each  aircraft  at  each 
boundary  size,  and  are  plotted  in  figure  9.  The  data  show  that  for  each  aircraft, 
aggressiveness  increased  as  the  boundaries  decrease.  This  mirrored  the  results  shown  in 
figure  8  above,  and  further  confirmed  the  theory  that  boundaries  change  pilot  behavior. 
The  duty  cycle  also  increased  as  boundaries  decreased  for  Aircraft  X  and  Y.  However, 
duty  cycle  was  relatively  constant  with  boundary  size  for  Aircraft  Z.  This  was  likely  due 
to  the  poor  handling  qualities  of  Aircraft  Z,  requiring  the  pilot  to  “stay  out  of  the  loop”  as 
much  as  possible  in  order  to  maintain  appropriate  control.  Also  note  that  the  decrease  in 
duty  cycle  and  aggressiveness  for  Aircraft  Y  in  LAMARS  at  the  10.5  degree  boundary 
size  was  due  to  the  fact  that  only  three  pilots  were  able  to  operate  to  this  boundary  size, 
and  their  duty  cycle  and  aggressiveness  was,  on  average,  lower  than  the  other  pilots. 
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Figure  9:  Average  Duty  Cycle  and  Aggressiveness  (Average  of  All  Runs) 


In  addition  to  affecting  inceptor  movement,  boundaries  also  affected  the  pilot’s 
tracking  performance.  With  large  boundaries,  the  required  pilot  performance  was  low. 
As  the  boundaries  shrank,  the  required  perfonnance  increased,  and  the  tracking 
perfonnance  subsequently  increased  (i.e.  the  roll  error  decreased),  even  though  the 
tracking  task  had  not  changed.  Figure  10  shows  the  general  trend  of  increasing 
performance  as  a  function  of  decreasing  boundary  size. 
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Figure  10:  Average  Roll  Error  (Average  of  All  Runs) 


Additional  insight  was  gained  by  plotting  the  roll  error  as  a  percentage  of 
boundary  size.  While  the  absolute  error  (figure  10)  decreased  as  a  function  of  boundary 
size,  the  normalized  error  (figure  11)  remained  roughly  constant  until  the  boundaries 
decreased  to  a  given  size,  at  which  point  the  normalized  error  began  to  increase  and 
performance  decreased.  From  these  data,  it  can  be  concluded  that  increasing 
performance  requirements  increased  performance  until  the  requirements  were  too 
stringent,  at  which  point  the  increasing  requirements  actually  decreased  performance. 
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Normalized  Average  Roll  Error 
Altitude:  20,000  ft  PA  Airspeed:  300  KIAS 


Aircraft:  LAMARS 


Dates:  16-18  Jul  2008 


Figure  11:  Normalized  Roll  Error  (Average  of  All  Runs) 
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Another  objective  of  the  AT  BAT  project  was  to  quantitatively  model  a  pilot’s 
behavior  in  the  presence  of  boundaries.  The  proposed  BAT  pilot  model  was  a  switching 
model,  in  which  a  pilot  engaged  in  point  tracking  (i.e.  making  inputs  solely  in  response  to 
tracking  error)  until  boundaries  became  a  factor  (i.e.  until  the  pilot  felt  that  boundaries 
were  likely  to  be  exceeded).  When  boundaries  became  a  factor,  the  pilot  in  the  proposed 
model  began  tracking  the  boundaries  by  making  inputs  solely  to  avoid  the  boundaries 
(and  neglecting  the  point  tracking  task).  In  this  model,  the  pilot  was  described  by  table  1 
(reference  1): 


Table  1:  BAT  Model  Description 


Situation 

Boundary  Awareness 

Boundary  Feedback 

Displacement  inside, 
moving  away  from  boundary 

No  threat 

0 

Displacement  inside, 
moving  toward  boundary 

tb  —  tmin 

No  threat 

0 

tmax  tb  <  tmin 

Feedback  increases  linearly  as  tb 
descreases 

tmin  —  (.tb  f  Tii)  lyr 
.  .  & bm 

^min  tmax 

where 

tb  =  xb(dxb/dt)~ 1 

tb  —  tmax 

Maximum  threat 

Khm. 

Displacement  outside  boundary 

Maximum  threat 

Khm. 

Where: 

tmin:  The  minimum  time  to  boundary  at  which  the  pilot  first  began  reacting  to  the 
boundaries;  a  larger  value  of  tmin  implies  that  the  pilot  begins  reacting  to 
boundaries  sooner 

tmax:  The  time  at  which  the  pilot  input  the  maximum  force  to  avoid  a  boundary;  a 

larger  value  of  tmax  implies  that  the  pilot  applies  maximum  force  sooner 

h,:  The  time  to  boundary  based  on  the  boundary  size  and  the  pilot’s  error  and  roll 

rate 

Tt,:  Pilot  time  delay 

Kbriv  The  max  stick  force  the  pilot  input 
xb:  Distance  to  boundary 
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Equation  1  in  the  table  described  pilot  inputs  only  during  periods  of  boundary 
tracking.  When  boundaries  were  not  a  factor,  the  BAT  equation  predicted  zero  force.  At 
this  point,  a  point  tracking  model  would  need  to  be  used  to  predict  point  tracking  force. 
For  this  test,  a  time  history  of  the  pilot’s  stick  force  inputs  was  plotted  against  a  BAT 
prediction  of  the  inputs.  The  values  were  tuned  until  there  was  agreement  between  what 
the  pilot  actually  did  in  the  test  and  what  the  BAT  model  would  have  predicted.  The 
tuning  was  an  iterative  process  whereby  the  values  for  each  of  the  variables  described  in 
table  1  were  changed  in  small  increments  until  some  combination  of  all  the  values 
resulted  in  a  predicted  input  that  matched  the  actual  pilot  input.  Only  periods  of 
suspected  boundary  avoidance  (from  pilot  comments  or  observed  behavior)  were 
analyzed  for  this  prediction.  Example  results  from  a  LAMARS  profile  and  a  flight 
profile  in  the  VISTA  are  shown  in  figures  12  and  13. 


LAMARS  Pilot  Input  and  BAT  Predicted  Input  Time  History 
Altitude:  20.000  ft  PA  Airspeed:  300KIAS 
Aircraft:  LAMARS  Dates:  15-18  July 
Project:  AT  BAT 


Figure  12:  LAMARS  Pilot  Input  and  BAT  Predicted  Input  Time  History 
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VISTA  Pilot  Input  and  BAT  Predicted  Input  Time  Histories 
Altitude:  20,000  ft  PA  Airspeed:  300  KIAS 
Aircraft:  VISTA  Dates:  10-23  Sep  2008 

Project:  AT  BAT 


Figure  13:  VISTA  Pilot  Input  and  BAT  Predicted  Input  Time  History 

The  model  tuning  was  accomplished  for  every  BAT  and  BAT  plus  secondary  task 
profile  flown  in  both  LAMARS  and  the  VISTA,  for  a  total  of  137  profiles.  The  ranges  of 
values  that  resulted  in  good  agreement  between  actual  and  predicted  inputs  are  shown  in 
table  2. 


Table  2:  BAT  Prediction  Values 


LAMARS 

VISTA 

train  (seconds) 

min 

2 

1.3 

max 

5 

5 

tmax  (seconds) 

min 

0.06 

0.06 

max 

0.1 

0.4 

Tb  (seconds) 

min 

0.2 

0.15 

max 

0.25 

0.25 

Kbm  (pounds) 

min 

10 

11 

max 

28 

19 

There  are  a  few  interesting  items  to  note  from  the  table  and  plots  of  the  BAT 
predictions.  First,  there  was  generally  better  agreement  between  the  prediction  and  actual 
inputs  in  the  LAMARS  data  for  nearly  the  entire  time  history,  even  when  it  could  be 
argued  the  pilot  was  acting  only  to  follow  the  roll  task  and  was  not  close  enough  to  a 
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boundary  to  be  exhibiting  boundary  avoidance  behavior.  In  the  VISTA  comparisons,  it 
was  only  possible  to  get  acceptable  agreement  where  the  pilot  was  clearly  exhibiting 
boundary  avoidance  behavior.  This  was  defined  as  any  point  in  the  time  history  where 
the  pilot  exceeded  or  came  close  to  exceeding  a  boundary  and/or  where  he  exhibited 
pilot-in-the-loop  oscillations  (PIOs).  PIOs  were  determined  as  any  point  where  the 
pilot-commanded  roll  angle  oscillated  around  the  roll  task  angle  for  more  than  two 
cycles.  It  was  at  these  points  that  the  BAT  prediction  parameters  were  tuned  to  match  the 
actual  pilot  inputs.  Given  that  the  pilot  had  less  to  split  his  time  and  focus  on  in  the 
simulator  than  the  actual  flight  test,  it  is  possible  that  the  addition  of  the  real-world 
variables  introduced  in  the  flight  test  sorties  caused  larger  differences  between  the  model 
and  the  actual  outputs. 

Figure  14  shows  the  range  of  values  for  each  of  the  BAT  model  parameters  for 
both  the  VISTA  flights  and  the  LAMARS  data.  Both  sets  of  data  included  three  different 
aircraft,  trying  to  model  three  different  levels  of  handling  qualities.  This  resulted  in  six 
different  aircraft  models  between  the  LAMARS  and  VISTA  sorties. 

Overall,  the  tm;n  values  in  the  VISTA  sorties  were  shifted  lower  than  in  the 
LAMARS  data.  This  indicates  that  in  the  LAMARS  simulations  the  pilots  were  reacting 
to  boundaries  sooner  than  in  the  VISTA  sorties.  It  is  possible  that  the  pilot  was 
experiencing  more  “seat-of-the-pants”  motion  in  flight  tests  than  in  the  simulator,  and  as 
a  result  was  busy  managing  other  tasks,  such  as  maintaining  altitude,  thereby  delaying 
any  reactions  to  boundaries  until  he  was  closer  to  the  boundaries.  This 
“seat-of-the-pants”  feeling  would  also  explain  why  there  was  a  larger  spread  and  higher 
overall  value  for  tmax  in  the  VISTA  data  than  in  the  LAMARS  data.  The  higher  tmax 
indicates  that  the  pilot  was  inputting  maximum  force  even  when  he  was  farther  from  the 
boundaries  than  in  the  LAMARS  data.  The  true  motion  of  the  aircraft  was  a  better 
indicator  than  even  a  motion-based  simulator  for  such  parameters  as  the  rate  he  was 
approaching  a  boundary,  which  would  explain  why  he  was  reacting  with  maximum  force 
to  a  boundary  sooner  than  in  the  LAMARS  data. 

Additionally,  there  was  a  greater  range  of  pilot  delay  time  (ib)  in  the  VISTA  data. 
The  motion  of  the  aircraft  was  likely  indicating  to  the  pilot  sooner  that  he  needed  to  react, 
which  would  explain  why  in  some  cases  the  pilot  delay  was  shorter  than  in  LAMARS. 
There  was  a  tighter  grouping  of  max  force  (Kbm)  in  the  VISTA  data  as  well.  This  again 
was  likely  due  to  a  stronger  motion  cue  in  flight  data  than  in  the  simulator,  allowing  the 
pilot  to  shape  his  inputs  more  precisely  in  the  VISTA  than  in  the  LAMARS  data. 
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Comparison  of  LAMARS  and  VISTA  tmin  Values 

Altitude:  20,000  ft  PA  Dates:  15-18  July  2008, 10-23  Sep  2008 

Airspeed:  300KIAS  Project:  AT  BAT 
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Comparison  of  LAMARS  and  VISTA  tmax  Values 

Altitude:  20,000ft  PA  Dates:  15-18 July 2008, 10-23 Sep 2008 

Airspeed:  300KIAS  Project:  AT  BAT 


Comparison  of  LAMARS  and  VISTA  tb  Values 

Altitude:  20,000  ft  PA  Dates:  15-18  July  2008, 10-23  Sep  2008 

Airspeed:  300KIAS  Project:  AT  BAT 
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Figure  14:  BAT  Model  Parameter  Values 


In  summary,  the  following  conclusions  were  made  from  observing  pilot  behavior 
in  the  presence  of  boundaries.  Boundaries  changed  pilot  behavior.  As  boundaries 
decreased,  average  pilot  stick  force,  aggressiveness,  and  duty  cycle  increased,  even 
though  the  tracking  task  did  not  change.  Shrinking  boundaries  resulted  in  smaller  roll 
error  (i.e.  better  performance)  until  the  boundaries  became  too  small.  At  this  point,  the 
error  (as  a  percentage  of  boundary  size)  began  to  increase.  The  proposed  mathematical 
BAT  model  was  in  good  agreement  with  pilot  behavior  during  periods  of  boundary 
avoidance. 


Data  Analysis:  Cooper-Harper  Ratings 

The  second  objective  of  the  AT  BAT  program  was  to  determine  the  correlation 
between  performance  and  workload  (quantitatively  measured  during  a  BAT  plus 
secondary  task  profile)  and  the  Cooper-Harper  rating  of  a  given  aircraft  (qualitatively 
assigned  during  a  Cooper-Harper  profile).  BAT  performance  for  a  given  aircraft  model 
was  defined  as  the  average  BAT  task  run  time  achieved  with  that  model.  (BAT 
performance  could  have  also  been  defined  as  the  minimum  achievable  boundary  size. 
However,  due  to  a  small  handling  qualities  spread  among  aircraft  models  in  this  test,  this 
difference  in  BAT  task  run  time  was  small,  and  each  model  was  able  to  operate  to  the 
same  minimum  boundary  size.)  BAT  workload  was  defined  as  the  boundary  size  at 
which  the  average  secondary  task  perfonnance  began  to  degrade. 
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Histograms  of  Cooper-Harper  ratings  are  shown  in  figure  15.  As  described  in  the 
discussion  of  aircraft  model  development,  the  handling  qualities  spread  among  the 
aircraft  models  was  not  as  large  as  desired.  All  of  the  VISTA  models  and  two  of  the 
LAMARS  models  received  mostly  Level  2  ratings.  However,  some  small  differences 
among  the  aircraft  could  be  seen.  For  example,  in  VISTA  both  Aircraft  X  and  Aircraft  Y 
had  median  Cooper-Harper  ratings  of  4  (Level  2).  However,  Aircraft  X  received  two 
Level  1  ratings,  whereas  Aircraft  Y  did  not  receive  any  Level  1  ratings.  Aircraft  Z  had  a 
median  Cooper-Harper  rating  of  6.  In  LAMARS,  the  Aircraft  X,  Y,  and  Z  had  median 
Cooper-Harper  ratings  of  5,  6,  and  7,  respectively.  Pilot  comments  confirmed  that  the 
handling  qualities  of  each  aircraft  were  in  fact  different. 


Aircraft  X  Cooper  Harper  Ratings  Aircraft  Y  Cooper  Harper  Ratings  Aircraft  Z  Cooper  Harper  Ratings 

Altitude:  20,000  ft  PA  Airspeed:  300  KIAS  Altitude:  20,000  ft  PA  Airspeed:  300  KIAS  Altitude:  20,000  ft  PA  Airspeed:  300  KIAS 

Aircraft:  VISTA  Dates:  10-23  Sep  2008  Aircraft:  VISTA  Dates:  10-23  Sep  2008  Aircraft:  VISTA  Dates:  1 0-23  Sep  2008 


Cooper  Harper  Rating  Cooper  Harper  Rating  Cooper  Harper  Rating 

Figure  15:  Cooper-Harper  Rating  Histograms 


In  figure  16  the  median  Cooper-Harper  rating  is  displayed  as  a  function  of 
average  BAT  task  run  time.  The  error  bars  display  the  minimum  and  maximum 
Cooper-Harper  ratings.  In  the  LAMARS  data,  the  average  BAT  run  time  was  shortest  for 
the  aircraft  with  the  worst  handling  qualities  (Aircraft  Z).  Aircraft  X  and  Y  had  very 
similar  average  BAT  run  times,  which  was  not  surprising  given  the  small  spread  in 
Cooper-Harper  ratings  between  the  two.  The  VISTA  data  further  supported  this 
correlation.  The  average  BAT  run  time  was  shortest  for  the  aircraft  with  the  worst 
handling  qualities  (Aircraft  Z)  and  longest  for  the  aircraft  with  the  best  handling  qualities 
(Aircraft  X).  The  difference  in  average  run  time  was  small,  however.  Had  the 
boundaries  decreased  in  finer  increments,  it  may  have  been  possible  to  see  an  even  larger 
spread  in  BAT  run  times  to  add  further  credence  to  this  observation.  BAT  run  time  is  a 
quantitative  measure  of  task  performance  since  a  longer  run  time  would  indicate  better 
performance  by  the  pilot.  The  data  portrayed  in  figure  16  indicated  that  a  Cooper-Harper 
rating  was  indirectly  proportional  to  the  BAT  run  time,  meaning  a  better  rated  aircraft 
(smaller  Cooper-Harper  rating)  had  a  longer  BAT  run  time. 
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Cooper  Harper  Rating  Correlated  to  BAT  Run  Time 
Altitude:  20,000  ft  PA  Airspeed:  300  KIAS 
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Cooper  Harper  Rating  Correlated  to  BAT  Run  Time 
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Figure  16:  Cooper-Harper  Ratings  and  Average  BAT  Task  Run  Time 


Recall  that  the  secondary  task  was  used  as  a  quantitative  measure  of  workload. 
As  expected,  the  average  secondary  task  error  increased  with  decreasing  boundary  size, 
because  the  pilot  was  devoting  more  attention  to  staying  within  the  smaller  boundaries 
than  perfonning  well  on  the  secondary  task.  This  correlation  held  true  regardless  of  the 
aircraft  handling  qualities,  but  the  point  at  which  performance  began  to  decrease  varied 
with  the  aircraft  handling  qualities.  For  example,  as  shown  in  figure  17,  aircraft  X  in  the 
VISTA,  which  had  the  best  handling  qualities,  showed  relatively  constant  secondary  task 
error  for  the  first  two  boundary  sizes,  and  then  a  sudden  increase  in  error  when  the 
boundaries  shrank  to  ±16  degrees.  Since  the  boundaries  decreased  step-wise,  the  exact 
boundary  size  at  which  increased  error  would  occur  (referred  to  as  the  critical  boundary 
size)  could  not  be  determined.  Therefore,  a  range  of  critical  boundary  sizes  was  given. 


Average  Secondary  Task  Error 
Altitude:  20,000  ft  PA  Airspeed:  300  KIAS 


Average  Secondary  Task  Error 
Altitude:  20,000  ft  PA  Airspeed:  300  KIAS 


Figure  17:  Average  Secondary  Task  Error 


Table  3  shows  the  critical  boundary  size  ranges  for  each  aircraft.  In  the  VISTA, 
Aircraft  X  and  Y  had  the  same  critical  boundary  size,  probably  due  to  the  similarity  in 
handling  qualities.  In  LAMARS,  Aircraft  Y  actually  had  a  smaller  critical  boundary  size 
than  Aircraft  X,  even  though  the  pilot  opinion  ratings  were  worse  for  Aircraft  Y.  It  could 
be  argued  that  the  Aircraft  Y  error  was  actually  increasing  from  the  start,  and  thus  the 
critical  boundary  size  for  Aircraft  Y  was  greater  than  32  degrees.  This  was  almost 
definitely  the  case  for  Aircraft  Z  in  both  the  LAMARS  and  VISTA,  in  which  the  critical 
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boundary  size  could  not  be  directly  determined,  because  the  secondary  task  error 
increased  linearly  from  the  start  of  the  task.  The  task  would  need  to  be  started  at  a  larger 
boundary  size  in  order  to  detennine  the  critical  boundary  size. 


Table  3:  Critical  Boundary  Size  Ranges 


LAMARS 

VISTA 

Aircraft  X 

25.6-32 

16-23 

Aircraft  Y 

20.5-25.6 

16-23 

Aircraft  Z 

>32 

>23 

To  more  completely  understand  how  the  primary  task  affected  the  secondary  task 
perfonnance,  the  secondary  task  should  be  evaluated  in  isolation  (i.e.  without  a  primary 
task).  In  future  tests,  perform  a  secondary  task  profile  with  no  primary  task  in 
order  to  provide  a  performance  baseline.  (Rl1) 

In  general,  the  aircraft  with  better  Cooper-Harper  ratings  had  a  smaller  critical 
boundary  size.  Because  an  aircraft  with  better  Cooper-Harper  ratings  had  handling 
qualities  that  made  the  primary  roll  tracking  task  easier,  the  pilot  could  devote  his 
attention  to  doing  well  on  the  secondary  task,  even  in  the  presence  of  smaller  boundaries. 
Secondary  task  performance  began  to  degrade  earlier  (i.e.  at  larger  boundaries)  for 
aircraft  with  poor  handling  qualities.  In  this  comparison,  the  data  showed  that  the 
Cooper-Harper  rating  was  proportional  to  the  critical  boundary  size,  meaning  a  better 
rated  aircraft  (smaller  Cooper-Harper  rating)  had  a  smaller  critical  boundary  size. 

The  following  conclusions  could  therefore  be  made  from  comparing  performance 
and  workload  (quantitatively  measured  during  a  BAT  plus  secondary  task  profile)  with 
the  Cooper-Harper  rating  of  a  given  aircraft  (qualitatively  assigned  during  a 
Cooper-Harper  profile).  Aircraft  with  better  (i.e.  smaller)  Cooper-Harper  ratings  tended 
to  fly  longer  in  the  presence  of  boundaries  than  aircraft  with  worse  (i.e.  higher)  Cooper- 
Harper  ratings.  Aircraft  with  better  Cooper-Harper  ratings  tended  to  have  degraded 
secondary  task  performance  (i.e.  increased  workload  and  diminished  spare  capacity)  at 
smaller  boundary  sizes  than  aircraft  with  worse  Cooper-Harper  ratings.  The  boundary 
size  at  which  the  secondary  task  perfonnance  degraded  was  called  the  critical  boundary 
size. 


In  order  for  a  Cooper-Harper  rating  to  be  relevant,  the  assigned  task  and  the 
perfonnance  requirements  were  required  to  be  operationally  representative.  Likewise, 
when  using  a  BAT  task  to  quantify  pilot  opinion,  the  given  task  and  the  requirements 
(desired  and  adequate  minimum  achievable  and  critical  boundary  sizes)  must  be  suitable 
for  the  aircraft  under  test.  For  this  test,  suitable  boundary  sizes  were  determined  through 


1  Numerals  preceded  by  an  R  within  parentheses  at  the  end  of  a  sentence  correspond  to  the  recommendation 
numbers  tabulated  in  the  Conclusions  and  Recommendations  section  of  this  report. 
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trial  and  error  during  simulations  and  calibration  flights.  The  amount  of  BAT  flight  test 
data  was  not  sufficient  to  pre-define  desired  and  adequate  boundary  sizes  for  all  types  of 
aircraft.  Further  BAT  testing  on  aircraft  with  varying  levels  of  handling  qualities  would 
allow  these  required  boundary  sizes  to  be  detennined  for  various  aircraft  classes  and 
flight  phase  categories,  much  the  same  way  as  MIL-STD-1797B  suitable  flying  qualities 
parameters  were  developed  through  historical  data.  Perform  more  testing  with 
boundaries  and  secondary  tasks  in  order  to  better  quantify  the  relationship  between 
subjective  workload  assessment  and  secondary  task  performance.  (R2) 

With  adequately  defined  performance  and  workload  criteria  (in  terms  of  desired 
and  adequate  minimum  achievable  and  critical  boundary  sizes),  a  chart  such  as  figure  18 
could  be  built,  in  which  the  critical  boundary  size  would  be  plotted  against  the  minimum 
achievable  boundary  size.  In  this  chart,  the  area  in  which  the  plotted  point  lay  would 
correspond  to  the  handling  qualities  level.  Aircraft  with  a  minimum  achievable  boundary 
size  and  a  critical  boundary  size  smaller  than  the  desired  sizes  would  be  Level  1 .  Aircraft 
with  either  (or  both)  minimum  achievable  boundary  size  or  a  critical  boundary  size 
between  the  desired  and  adequate  values  would  be  Level  2.  Aircraft  with  either  (or  both) 
minimum  achievable  boundary  size  or  a  critical  boundary  size  between  outside  of  the 
adequate  values  would  be  Level  3. 

Figure  18  plots  the  average  results  (discussed  in  the  section  above)  for  each 
aircraft  model  in  the  LAMARS  and  VISTA.  The  desired  and  adequate  boundary  sizes 
were  chosen  after  testing  so  that  the  results  corresponded  with  the  assigned  Cooper- 
Harper  rating  levels.  As  mentioned  above,  more  testing  (in  order  to  establish  a  large 
database  for  various  aircraft  classes  and  flight  phase  categories,)  would  allow  these 
values  to  be  pre-detennined.  In  the  chart,  all  models  but  one  fell  in  the  Level  2  region. 
Aircraft  Z  in  LAMARS  fell  in  the  Level  3  region.  This  corresponded  to  the  assigned 
Cooper-Harper  ratings. 
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BAT  Task  Performance  and  Workload 


Minimum  Achievable  Boundary  Size,  degrees  (Performance) 

Figure  18:  BAT  Performance  and  Workload  Correlated  to  Handling  Qualities  Levels 
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Conclusions  and  Recommendations 

The  objective  of  the  AT  BAT  program  was  to  observe  pilot  behavior  during  a 
tracking  task  and  secondary  task  in  the  presence  of  boundaries,  and  compare  this 
behavior  to  the  behavior  predicted  by  the  boundary  avoidance  tracking  (BAT)  theory.  A 
second  objective  was  to  search  for  a  correlation  between  the  subjective  pilot  opinion 
ratings  of  the  aircraft  from  a  Cooper-Harper  task  and  the  objectively-measured 
perfonnance  and  workload  of  a  separate  task  in  the  roll  axis.  All  objectives  were  met. 

Boundaries  changed  pilot  behavior.  As  boundaries  decreased,  average  pilot  stick 
force,  aggressiveness,  and  duty  cycle  increased,  even  though  the  tracking  task  did  not 
change.  Shrinking  boundaries  resulted  in  smaller  roll  error  (i.e.  better  perfonnance)  until 
the  boundaries  became  too  small.  At  this  point,  the  enor  (as  a  percentage  of  boundary 
size)  began  to  increase.  The  proposed  mathematical  BAT  model  qualitatively  matched 
pilot  behavior  during  periods  of  boundary  avoidance. 

Comparisons  were  made  between  performance  and  workload  (quantitatively 
measured  during  a  BAT  plus  secondary  task  profile)  with  the  Cooper-Harper  rating  of  a 
given  aircraft  (qualitatively  assigned  during  a  Cooper-Harper  profile).  Aircraft  with 
better  (i.e.  smaller)  Cooper-Harper  ratings  tended  to  fly  longer  in  the  presence  of 
boundaries  than  aircraft  with  worse  (i.e.  higher)  Cooper-Harper  ratings.  Due  to  a  small 
handling  qualities  spread  among  aircraft  models,  however,  this  difference  in  BAT  task 
run  time  was  small.  Aircraft  with  better  Cooper-Harper  ratings  tended  to  have  degraded 
secondary  task  performance  (i.e.  increased  workload  and  diminished  spare  capacity)  at 
smaller  boundary  sizes  than  aircraft  with  worse  Cooper-Harper  ratings.  The  boundary 
size  at  which  the  secondary  task  perfonnance  degraded  was  called  the  critical  boundary 
size. 


With  adequately  defined  performance  and  workload  criteria  (in  terms  of  desired 
and  adequate  minimum  achievable  and  critical  boundary  sizes),  critical  boundary  size 
could  be  plotted  against  the  minimum  achievable  boundary  size.  The  position  of  the 
plotted  point  would  correspond  to  the  handling  qualities  level.  Aircraft  with  a  minimum 
achievable  boundary  size  and  a  critical  boundary  size  smaller  than  the  desired  sizes 
would  be  Level  1 .  Aircraft  with  either  (or  both)  minimum  achievable  boundary  size  or  a 
critical  boundary  size  between  the  desired  and  adequate  values  would  be  Level  2. 
Aircraft  with  either  (or  both)  minimum  achievable  boundary  size  or  a  critical  boundary 
size  between  outside  of  the  adequate  values  would  be  Level  3. 

The  following  recommendations  were  made: 

In  future  tests,  perform  a  secondary  task  profile  with  no  primary  task  in  order  to 
provide  a  performance  baseline.  (Rl,  p.23) 

Perform  more  testing  with  boundaries  and  secondary  tasks  in  order  to  determine 
desired  and  adequate  minimum  achievable  and  critical  boundary  sizes  for  various 
aircraft  classes  and  flight  phase  categories.  (R2,  p.  24) 
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APPENDIX  A  -  AIRCRAFT  MODEL  DESCRIPTIONS 


The  aircraft  models  in  LAMARS  had  the  following  characteristics: 


Aircraft  X: 


p  _  —29.15s3— 47.07s2— 242. 5S+0.45  76 
8a  ~  s4 +3.8 Is3  +  12.64s2  + 18. 13S+0.181 


a)dr  =  3  radians /second 


(dr  —  0-3 


rr  =  0.5  seconds 


ts  —  99  seconds 


Aircraft  Y : 


p  _  —11.72s3— 2.406s2  — 10. 44s +0.0 1966 

8  a  s4  + 1. 1 19s3  +  1.193  s2  +0.9209S+0.009254 


a)dr  —  1  radians /second 

(dr  =  0.1 

rr  =  1.1  seconds 


rs  =  98  seconds 


Aircraft  Z: 


p  _  —6.152s3— 0.2133s2— 0.8 743S+0.00 1645 

8  a  s4+0. 526s3 +  0.1 732s2  +0.08 168s +0.000803  3 


a)dr  —  0 .4  radians / second 


(dr  —  0.02 

rr  =  2.5  seconds 


rs  =  100  seconds 


Where: 

p  =  roll  rate 

5a  =  aileron  deflection  (degrees) 
o>dr  =  Dutch  roll  frequency 
/dr  =  Dutch  roll  damping 
rr  =  roll  mode  time  constant 
rs  =  spiral  mode  time  constant 
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The  roll  mode  and  Dutch  roll  parameters  were  chosen  from  MIL-STD-1797B 
(reference  4)  to  correspond  with  Level  1,  2,  and  3  handling  qualities.  Due  to  the 
continuously  closed-loop  nature  of  the  roll  tracking  task  in  this  project,  the  spiral  mode 
was  not  considered  a  factor,  and  was  not  changed  significantly  among  the  models. 

The  stick  force  gradient  was  nonlinear,  and  was  based  on  an  F-16  gradient. 
Therefore,  the  transfer  functions  are  presented  here  as  a  function  of  aileron  input  instead 
of  stick  force.  Figure  A-l  shows  the  step  responses  of  each  of  the  LAMARS  aircraft 
models.  The  quick  rise  time  and  high  Dutch  roll  damping  are  evident  in  the  Aircraft  X 
response.  The  Aircraft  Y  and  Aircraft  Z  responses  showed  progressively  slower  rise 
time,  and  lower  Dutch  roll  frequency  and  damping. 


Time,  sec 

Figure  A-  1:  Roll  rate  response  to  aileron  step,  LAMARS 


Note  that  all  of  the  transfer  functions  contained  a  very  slow  non-minimum  phase 
zero.  Due  to  the  chosen  sign  convention,  this  manifested  itself  in  an  initial  response  in 
the  desired  direction,  followed  by  a  very  slow  decay  to  a  steady-state  response  in  the 
opposite  direction.  Figure  A-2  shows  the  Aircraft  X  long-term  response.  Aircraft  Y  and 
Z  had  similar  long-tenn  responses.  Due  to  the  very  slow  nature  of  this  behavior,  it  was 
not  noticed  by  the  pilots. 


A-2 


December  2008 


AT  BAT 


LAMARS  Aircraft  X  predicted  long-term  roll  rate  response  to  a  step  aileron  input 
1  5r 


0  100  200  300  400  500  600 

Time,  sec 

Figure  A-  2:  Long  term  roll  rate  response  to  aileron  step,  LAMARS 


Because  the  handling  qualities  spread  among  the  LAMARS  models  was  less  than 
desired,  the  models  were  changed  for  implementation  in  the  VISTA.  After  several 
iterations  (including  results  from  the  calibration  sorties),  the  team  decided  to  design  each 
model  with  Level  1  Dutch  roll  characteristics,  and  use  the  roll  mode  time  constant  as  the 
discriminator  between  the  models.  The  spiral  mode  time  constant  was  not  intentionally 
changed,  but  it  was  allowed  to  change  as  necessary  to  appropriately  model  the  roll  mode 
in  the  VISTA.  The  following  models  were  used  for  the  VISTA  flight  test: 


Aircraft  X: 


p  _  —41.02s3  — 73.38s2— 403.75+0.7464 
Sa  ~  s4  +  5.157s3  +  15.13s2+30.49s+0.9795 


c odr  —  3  radians  /second 


(dr  ~  0-3 

rr  =  0.3  seconds 


ts  —  31  seconds 


Aircraft  Y : 


p  _  —13.05s3  — 26.96s2  — 122. Is+O.2 2 52 

8  a  s4 +2.72  2s3  + 10.68s2 +8. 424s+ 0.1631 


c odr  —  3  radians  /second 


(dr  ~  0-3 

rr  =  1.1  seconds 


ts  —  50  seconds 
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Aircraft  Z: 


p  _  —10.31s3  — 17.06s2— 95.53s +0.1 773 

8  a  s4 +2.2 13s3 +9.7  53s2 +3.804S— 0.00801 


c odr  —  3  radians /second 

(dr  ~  0-3 

rr  =  2.4  seconds 


td  —  380  seconds  ( time  to  double  amplitude ) 

Figure  A-3  shows  the  predicted  step  responses  of  each  of  the  VISTA  aircraft 
models.  The  quick  rise  time  is  evident  in  the  Aircraft  X  response.  The  Aircraft  Y  and 
Aircraft  Z  responses  show  progressively  slower  rise  time.  All  models  exhibited 
acceptable  Dutch  roll  parameters.  Note  that  the  predicted  spiral  mode  for  Aircraft  Z  was 
unstable.  The  VISTA  predicted  models  also  displayed  the  same  non-minimum  phase 
behavior  as  the  LAMARS  models. 


VISTA  Aircraft  X  predicted  roll  rate  response  to  a  step  aileron  input  VISTA  Aircraft  Y  predicted  roll  rate  response  to  a  step  aileron  input 
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VISTA  Aircraft  Z  predicted  roll  rate  response  to  a  step  aileron  input 


Figure  A-  3:  Predicted  roll  rate  response  to  aileron  step,  VISTA 


During  flight  test,  lateral  step  inputs  were  performed  in  order  to  validate  the 
predicted  response  displayed  in  figure  A-3.  The  step  inputs  were  performed  with  lateral 
stick  inputs  instead  of  pure  aileron  inputs.  Since  the  stick  force  gradient  was  nonlinear,  it 
couldn’t  be  adequately  modeled  with  a  transfer  function.  Therefore,  the  predicted 
aileron-to-roll-rate  response  and  the  flight  test  stick-to-roll-rate  response  can’t  be  directly 
compared.  However,  the  results  can  be  compared  qualitatively  and  are  presented  in 
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figure  A-4.  Note  that  the  roll  mode  time  constants  are  significantly  higher  than 
predicted.  In  fact,  the  Aircraft  Y  response  never  reached  steady  state  (with  a  3  second 
input).  The  Aircraft  Z  response  was  unstable,  with  a  much  larger  time-to-double 
amplitude  than  predicted. 


VISTA  Aircraft  X  flight  test  roll  rate  response  to  a  step  aileron  input  VISTA  Aircraft  Y  flight  test  roll  rate  response  to  a  step  aileron  input 


VISTA  Aircraft  Z  flight  test  roll  rate  response  to  a  step  aileron  input 


Figure  A-  4:  Flight  test  roll  rate  response  to  aileron  step,  VISTA 

In  summary,  the  actual  aircraft  characteristics  were  not  as  predicted,  but  were 
suitable  for  flight  test  use. 
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APPENDIX  B  -  COMPLETED  TEST  POINT  MATRIX 

Table  4:  Completed  Test  Point  Matrix 


Test 

Pilot 

Profile 

Aircraft  Model 

1 

1 

BAT  Plus 

X 

2 

1 

BAT 

Y 

3 

1 

BAT  Plus 

Z 

4 

1 

CHR 

Y 

5 

1 

BAT 

X 

6 

1 

CHR 

Z 

7 

1 

BAT  Plus 

Y 

8 

1 

BAT 

Z 

9 

1 

CHR 

X 

2 

CHR 

Y 

2 

BAT 

X 

El 

2 

CHR 

z 

13 

2 

BAT  Plus 

X 

14 

2 

BAT 

Y 

15 

2 

BAT  Plus 

z 

16 

2 

CHR 

X 

17 

2 

BAT 

z 

18 

2 

BAT  Plus 

Y 

3 

BAT 

X 

3 

CHR 

Y 

3 

BAT 

z 

3 

BAT  Plus 

Y 

3 

CHR 

X 

3 

BAT  Plus 

z 

3 

BAT 

Y 

3 

CHR 

Z 

3 

BAT  Plus 

X 

4 

BAT  Plus 

X 

4 

BAT 

Y 

4 

BAT  Plus 

z 

4 

CHR 

Y 

4 

BAT 

X 

4 

CHR 

z 

4 

BAT  Plus 

Y 

4 

BAT 

Z 
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Test 

Pilot 

Aircraft  Model 

36 

4 

X 

ca 

5 

BAT 

Z 

5 

BAT  Plus 

X 

5 

BAT 

Y 

5 

CHR 

X 

5 

BAT  Plus 

z 

5 

CHR 

Y 

5 

BAT 

X 

5 

BAT  Plus 

Y 

5 

CHR 

Z 

1 

BAT  Plus 

X 

1 

CHR 

z 

1 

BAT  Plus 

Y 

1 

BAT 

z 

1 

CHR 

X 

1 

BAT 

Y 

1 

BAT  Plus 

z 

1 

CHR 

Y 

1 

BAT 

X 

2 

BAT 

Y 

2 

BAT  Plus 

X 

2 

BAT 

Z 

2 

CHR 

X 

2 

BAT  Plus 

Y 

2 

CHR 

Z 

2 

BAT 

X 

2 

BAT  Plus 

z 

2 

CHR 

Y 

3 

BAT 

X 

3 

BAT  Plus 

Y 

3 

BAT 

Z 

3 

CHR 

Y 

3 

BAT  Plus 

X 

3 

CHR 

Z 

3 

BAT 

Y 

3 

BAT  Plus 

Z 

3 

CHR 

X 

4 

BAT 

z 

El 

4 

CHR 

X 
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Test 

Pilot 

Profile 

Aircraft  Model 

75 

4 

BAT 

Y 

76 

4 

BAT  Plus 

X 

77 

4 

CHR 

Z 

78 

4 

BAT  Plus 

Y 

79 

4 

BAT 

X 

80 

4 

CHR 

Y 

81 

4 

BAT  Plus 

Z 

82 

5 

CHR 

Y 

83 

5 

BAT 

X 

84 

5 

CHR 

Z 

85 

5 

BAT  Plus 

X 

86 

5 

BAT 

Y 

87 

5 

BAT  Plus 

Z 

88 

5 

CHR 

X 

89 

5 

BAT 

z 

90 

5 

BAT  Plus 

Y 

CHR  -  Roll-angle  tracking  task  with  no  boundaries  or  secondary  task 

BAT  -  Roll-angle  tracking  task  with  boundaries  but  no  secondary  task 

BAT  Plus  -  Roll-angle  tracking  task  with  boundaries  and  a  secondary  task 
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APPENDIX  C  -  COOPER-HARPER  RATING  SCALE 
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APPENDIX  D  -  ACRONYM  LIST 


Symbol  Definition  Units 

AFFTC  Air  Force  Flight  Test  Center 

AFIT  Air  Force  Institute  of  Technology 

BAT  boundary  avoidance  tracking 

CCD  charge-coupled  device 

CHR  Cooper-Harper  rating 

EP  evaluation  pilot 

FCP  front  cockpit 

HUD  heads-up  display 

LAMARS  Large  Amplitude  Multi-Mode  Aerospace  Research  Simulator 

PIO  pilot-in-the-loop  oscillation 

RCP  rear  cockpit 

SP  safety  pilot 

TMP  Test  Management  Project 

TPS  U.S.  Air  Force  Test  Pilot  School 

USAF  United  States  Air  Force 

VISTA  Variable-Stability  In-Flight  Simulator  Test  Aircraft 

VSS  variable  stability  system 
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